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*This We Believe Characteristics 

• High expectations for every member of the learning community 

• Students and teachers engaged in active learning 

• Assessment and evaluation programs that promote quality learning 

* Denotes the corresponding characteristics from NMSA's position paper, This We Believe, for 
this article. 


Heidi Andrade, Colleen Buff, J oe Terry, Marilyn Erano, & Shaun Paolino 

One lesson I've learned is that this is a process; it does not change students' 
writing overnight. But if you make it a continuous effort and incorporate it in all the 
writing you complete, the students will slowly develop their writing skills and their 
writing will improve. The most valuable lesson I learned is that students really do 
want to be successful and can rise to a challenge. (Mrs. Buff, eighth grade ELA 
teacher) 

In the fall of 2005 the principal and teachers at Knickerbacker Middle School (KMS) were worried. 
KMS was a "School in Good Standing" but had not hit federal or state benchmarks because of low 
subgroup scores on the English Language Arts (ELA) test— especially scores received by 
economically disadvantaged students, about half of this urban school's student population. KMS 
would be identified as a "School in Need of Improvement" in 2006 if the ELA scores did not 
improve. This article chronicles a successful attempt by the authors and their colleagues to teach 
writing by making improvements in the assessment of writing in the classroom. 

Our goals: What we attempted and why 

Our work together began in the fall of 2005, when Shaun Paolino, the principal, invited Heidi 
Andrade to help improve students' writing skills and scores. To meet the overarching goal of 
improving the assessment of writing at KMS, Prof. Andrade collaborated with the sixth, seventh, 
and eighth grade teachers of English and social studies. We set three goals: 

1. Make assessment processes, criteria, and standards crystal clear to students. 

2. Provide frequent, useful feedback to students about the quality of their work via teacher, 
peer, and self-assessment. 

3. Use the assessments to analyze the strengths and weaknesses in students' work and to 
plan instruction. 

These three goals are grounded in the literature on formative assessment. Most people think of 
assessment as the test at the end of a unit that tells teachers whether or not students "got it." 
That is a summative view of assessment, and tells only part of the story. Formative assessments 
happen before and while students work on assignments. A significant element of effective 
classroom assessment is formative— the kind of ongoing, regular feedback about student work 
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that leads to adjustment and revision by both the teacher and the students (Center for 
Educational Research & Innovation, 2005). 

A formative conception of assessment honors the crucial role of feedback in the development of 
understanding and skill building. This perspective on assessment is common in sports and in the 
arts, where students expect and receive frequent comments from coaches and directors about 
their performance (White, 1998). However, in spite of research that shows that feedback 
promotes learning and achievement (Black & Wiliam, 1998; Butler & Winne, 1995; Chappuis, 
2005), many students get little informative feedback about their work. Often, this is because few 
teachers have the luxury of regularly responding to each student's work and learning. 

Fortunately, research shows that students themselves can be useful sources of feedback via peer 
and self-assessment (Andrade & Boulay, 2003; Andrade, Du, & Wang, 2008; O'Donnell & 

Topping, 1998). 

Peer and self-assessment are key elements in formative assessment, because they involve 
students in thinking about the quality of their own and each others' work, rather than relying on 
their teachers as the sole source of evaluative judgment. There are many ways to scaffold 
effective peer and self-assessment. Self-assessment can be as simple as students circling the text 
on a rubric that best describes their work and attaching the marked-up rubric to the assignment 
before handing it in (Andrade & Boulay, 2003). Peer assessment is often done by giving rubric- 
referenced verbal feedback in class (O'Donnell & Topping, 1998). Regardless of how it is done, 
neither the peer nor the self-assessments count toward final grades, because this is formative, 
not summative, assessment. 

Our work at KMS taught us that formative assessment can play a key role in helping students 
learn to write. Predictably, however, assessment was not a silver bullet; We had to define and 
address other important issues before students could not only learn but also demonstrate what 
they had learned on the ELA test. 

The process: What we did and how 

The collaboration between a university consultant and Knickerbacker Middle School teachers 
began after a brief workshop in October of 2005 that introduced the notion of assessment as a 
moment of learning, research on the ways in which feedback can promote learning and 
achievement, and rubric- referenced assessment techniques, including peer and self-assessment. 
Monthly meetings started in November. The meetings were attended by the co-authors of this 
article and six other teachers, including one special education teacher. Held after school from 
2; 15 to 3; 30, the meetings initially focused on finding or developing rubrics for writing and using 
them with students. 





Designing a common rubric for writing 

In February of 2006 the team decided to create consistency 
across classes and grade levels by designing a common 
writing rubric. Drawing on the New York State standards for 
English/Language Arts and the 6 -F 1 Traits of Writing 
(Northwest Regional Educational Laboratory, 2008; see 
Figure 1), we developed two rubrics (Figures 2 & 3). The 
rubrics are nearly identical except for slightly more 
sophisticated standards for sentence fluency and word 
choice on the eighth grade version. The sixth grade rubric is 
used in grade six and in the first half of the seventh grade 
year. The eighth grade rubric is used in the second half of the seventh grade year as well as in 
eighth grade. J oe Terry, one of the sixth grade teachers, describes how he began using the rubric 
with his students; 
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I realized that it would not be that difficult to adapt my teaching to the rubrics. I 
was using a 4, 3, 2, 1 method, which I had been trained to use to evaluate social 
studies essays, and I had already adapted that method to my ELA instruction. The 
new rubric had more categories than I had used before. As I began experimenting 
with it, I only used some of the categories. I left off voice and word choice and 
decided to concentrate on the organization of the essay. After a couple of tries, I 
added voice and word choice. 

Figure 1 

The 6 + 1 Traits of Writing®, excerpted from the NWREL website ( http://www.nwrel.org ) 

I deas: The ideas are the heart of the message, the content of the piece, the main theme, 
together with all the details that enrich and develop that theme. 

Organization: Organization is the internal structure of a piece of writing, the thread of central 
meaning, the pattern, so long as it fits the central idea. 

Voice: Voice is the writer coming through the words, the sense that a real person is speaking to 
us and cares about the message. It is the heart and soul of the writing, the magic, the wit, the 
feeling, the life and breath. 

Word Choice: Word choice is the use of rich, colorful, precise language that communicates not 
just in a functional way, but in a way that moves and enlightens the reader. 

Sentence Fiuency: Sentence fluency is the rhythm and flow of the language, the sound of 
word patterns, the way in which the writing plays to the ear, not just to the eye. 

Conventions: Conventions are the mechanical correctness of the piece— spelling, grammar and 
usage, paragraphing (indenting at the appropriate spots), use of capitals, and punctuation. 


Figure 2 

Generic grade six writing rubric 

^ Figure 3 

Generic grade eight writing rubric 

Teaching peer and seif-assessment 

Knowing that simply handing out rubrics would not magically produce good writers and high test 
scores, we concerned ourselves with the matter of engaging students in carefully considering the 
strengths and weaknesses of their works in progress, according to the standards set in the 
rubrics. During the monthly meetings we shared approaches to peer and self-assessment. 

Meghan D'Adamo, a sixth grade teacher, videotaped her students doing a "fishbowl," in which two 
students gave and received feedback while Mrs. D'Adamo coached, and the rest of the class 
observed. The videotape was shared with other interested teachers. Prof. Andrade was also 
videotaped doing a demonstration of her favorite approach to self-assessment in a seventh grade 
classroom, during which she guided students in using colored pencils to determine which criteria 
on a rubric their drafts had and had not yet met (see Andrade & Boulay, 2003, for details). 

All of the teachers agreed to implement some form of peer and/or self-assessment in their own 
classrooms, according to their judgments about what would work best with their students. Colleen 
Buff, one of the eighth grade teachers, took the following approach: 

I began the school year by introducing the rubric to my eighth grade students and 
using it on every writing assignment we completed. At the beginning of each 
assignment, we would review each criterion and the specific aspects of the 
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assignment to be thinking about when writing. After students wrote rough drafts, 
we came back together as a ciass and began the process of seif-assessment. The 
process had to be scaffoided, but students began to deveiop the skiiis necessary to 
reaiiy iook at their writing and determine its strengths and weaknesses. 

I began the seif-assessment process with students receiving modei essay papers. 

Together we used the rubric and waiked through each modei essay. Students coior 
coded their rubric using coiored penciis, and then the ciass coior coded the modei 
essays. After working through modei essays severai times, the students did the 
same coior coding to their own essays. This technique aiiowed students to visuaiize 
which criteria they were strong on and the areas that couid be improved. Not oniy 
were students iooking at their own writing, determining strengths and weaknesses, 
but they were aiso enjoying it. 

For the first three writing assignments, we waiked through this process together, 
and then siowiy I gave up controi and iet the students work through the process on 
their own. After each new writing assignment, the writing I was receiving from 
students was better than the previous one. 

Mariiyn Erano, another eighth grade teacher, incorporated the writing rubric into a peer editing 
technique she had been using for years. 

Working in pairs or groups of three, students switched papers and edited each 
other's work. They use the COACH process in addition to the rubric. COACH is an 
acronym for Commend (offer praise). Observe (note ways in which their writing is 
simiiar to the writing they are editing). Ask (ask the writer questions about what he 
or she meant or intended). Consider (aiways be considerate of the writer's 
feeiings), and Heip (offer heip in a usefui way). 

Students responded weii to the feedback generated with this process. I have found 
that it works especiaiiy weii with the rubric. To no one's surprise, students seem 
more interested in each other's comments than in mine. They were surprisingiy 
wiiiing to revise. In the past, they wouid rather have their teeth puiied with a pair 
of rusty piiers than revise a paper. I now sense that the idea of revision may not 
have been pianted firmiy enough in the writing process. Without the rubric and the 
COACH process, they may not have fuiiy understood what specific changes to make 
as they revised, or how to make them. 

At each monthiy meeting the team discussed the teachers' experiments with the rubrics and with 
peer and seif-assessment, taiked about what did and did not work, and pianned next steps. 

Checking validity and reliability 

I n Aprii of 2006 the team turned its attention to the vaiidity and reiiabiiity of the assessments. We 
aiready knew that our rubrics refiected the New York State standards, so they passed one test of 
vaiidity. Another important quaiity to test was the rubrics' accessibiiity to students. We informaiiy 
poiied the students for their reactions to the rubrics. Students had a few questions about the 
meanings of some words, which their teachers addressed. Other than that, students toid us that 
they understood and vaiued the rubrics. The teachers' ciassroom observations of peer and seif- 
assessment sessions confirmed the students' ciaims. 

To examine the reiiabiiity of our assessments, we examined the simiiarities and differences in 
grading with the rubrics. Was everyone using the rubrics in ways that produced simiiar grades, or 
were we grading idiosyncraticaiiy and, perhaps, unfairiy? Lisa Puckey, a seventh grade teacher, 
voiunteered to bring copies of two students' biographicai essays on Maicoim X to a meeting. We 
each scored an essay independentiy and then compared scores. We were surprised to discover 
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that our scores tended to be close. Though we did not strive for perfect inter-rater reliability, 
given the nature of writing, minor changes to the wording and organization of the rubric resulted 
in an even better assessment tool. 

Using assessment results to plan instruction 

In the fall of 2006 we turned to our third goal, which was to use the information provided by the 
rubrics to evaluate trends in students' performances and guide instructional decisions. Mr. Terry 
proposed an approach that was elegant in its simplicity: 

I have always evaluated essays by looking for trends. With previous experience in 
item analysis, I devised a way to chart the class trends using our generic rubric. I 
simply charted the number of 4s, 3s, 2s, and Is received by my students for each 
criterion on the rubric. The first essay I looked at this way showed a weakness in 
paragraphing for that particular class. Seeing this trend helped me to design a 
group lesson based on the common weakness. That way, no student needed to feel 
singled out. 

The other teachers adopted or adapted Mr. Terry's approach to pinpointing weaknesses in student 
writing. Mrs. Erano, for example, used peer assessment data to make on-the-spot decisions about 
targeted, short lessons, which motivated her classes to write better. 

I would ask students to raise their hands if they got a 1, 2, 3, or 4 for a certain 
criterion on the rubric. If the hands showed that a third or a quarter of the students 
needed improvement in paragraphing or organization, I demonstrated the way I 
would edit if I were peer editing. Students would give me additional suggestions to 
improve the example on the overhead, and I incorporated their suggestions in front 
of their eyes. 

An unexpected rivalry soon sprang up between two of my classes, and scores on 
the rubric became a way for them to win a contest. When I told one class that the 
other class did exceptionally well on paragraphing— "only three kids got threes and 
the rest all got fours!"— the news would not sit well with them. One class would 
work harder to "catch" the other class. Of course, I had to tell the winning class 
that they "owned" organization on that day, but they would have to keep working 
to stay on top. Being able to tally the scores quickly made the competition 
manageable. 

As a result of the teachers' experiments with rubric- referenced peer and self-assessment, our 
subsequent meetings focused on discussing weaknesses in students' writing and sharing 
resources designed to address them. 

Solving the transfer problem 

In November of 2006 Mrs. Buff made a distressing discovery: Although her eighth graders were 
writing more effectively when she scaffolded rubric- referenced formative assessment in the 
classroom, the quality of what they wrote under practice test conditions was very disappointing. 
This discovery was especially distressing because the state ELA test was coming up in J anuary. 
The team devoted a meeting to this problem and identified it as one of transfer: Though students 
could use the rubric to write well, they did not transfer their new skills to rubricless contexts. 
Because writing, in general, and the ELA test, in particular, happens in rubricless contexts, we 
were worried. 

We decided to address the transfer problem by teaching students to jog their memories by writing 
an acronym at the top of their papers before they began writing. With the students' help, we 
developed the following acronym for the criteria on the writing rubrics: Ideas, Organization, 
Paragraphs, Voice, Word choice. Sentences, and Conventions became lOPVWSC, which stood for I 
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Only Play Videogames While Snacking Chips. To reinforce the acronym, Mrs. Buff had her 
students snack on chips as they wrote. She also spoke with them about the transfer problem: 

The students and I had several discussions about scenarios In which they might be 
asked to write and how the rubric could be beneficial In each of those situations. 

Our discussion Included students looking at all the writing they had completed up to 
this point and filling out worksheets that outlined their strengths and weaknesses. 

This led to an In-depth discussion about why students were struggling with writing 
and what 1 could do to help them Improve. Students responded honestly to my 
questions: 

1. What Is so difficult about writing essays? 

o Finding Information to Include, 
o Too much writing. 

o Distractions In the room (people talking, It's too hot In the room), 
o Organizing my thoughts, 
o Why do we have to write essays anyway? 

2. Why don't you Include everything we go over. Including the criteria from the rubric? 

o My hand hurts when 1 write too much. 

o 1 don't feel like It. 

o I just want to get It done. 

o There Is too much Information to remember. 

o 1 think about what 1 want to write and not how to write It. 

o We're lazy. 

3. How can I help you? 

o Have us write essays once a week, 
o More quizzes on the different criteria areas, 
o Five minutes to let our brains rest before we write, 
o A quick review before each essay, 
o Have us review what we read, 
o Give us rewards for Improvement. 

4. What would help you remember all the criteria? 

o studying; review classes, 
o the I OPVWSC acronym, 
o more free writing time. 

The best response was this: 

"1 don't think there Is much more you can do for us. 1 think It Is us. If we are lazy, 
we have to just do It. I guess It all really depends on us." I loved that comment 
because It made me feel like the students noticed how much work had been done 
to help them, and now It was time for them to step up and make It work. But 1 
wondered how quickly they could shift gears. I truly believed each one of them 
could write amazing essays, but 1 still questioned how to get them to do It. 

Mrs. Buff decided to take her students' advice and give them rewards for making Improvements 
In their writing. We all knew that extrinsic rewards could undermine the Intrinsic reward of writing 
well, yet It was clear from the students' comments that they wanted and needed more motivation 
so the risk seemed small. Mrs. Buff set them a challenge: All students In the class had to receive 
a score of three (out of four) or higher on every criterion on the writing rubric. If they were 
successful they would receive one free Friday class. Including snacks, board games, and video 
games. In addition, the grade they received on the writing assignment would count as a test 
grade. Mrs. Buff describes the challenge as a roller-coaster ride of emotions: 
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The challenge began. The class was given two days to work on the writing 
assignment. On the first day they completed their outlines and rough drafts. Most 
students seemed positive and receptive to the challenge. One student, however, 
was struggling and very negative about the task before him. When I noticed his 
frustration, 1 pulled him Into the hallway for a quick chat. He communicated to me 
that he was too nervous to complete the task: He knew he was going to ruin It for 
the rest of the class. He stated, "There Is no way I can get a 3 on every criterion, 
yeah right, that will never happen, even If 1 really try." 

1 made a deal with him and told him It could be our little secret. Since this student 
struggled academically and had had difficulties with writing In the past, 1 thought It 
would be fair that. If he received a two for every criterion, the whole class could still 
receive the reward. He was relieved and felt confident that he could perform at a 
level two. As we entered the classroom, he sat down and started feverishly writing 
his essay. This was the most 1 had seen him write the whole year. 

1 collected everyone's rough drafts to preview before the following day's final draft 
session. That evening I went through the rough drafts and almost had a meltdown. 

I could not believe how much the students had forgotten to Include. I kept 
questioning whether I had been clear In my expectations. I could not figure out 
what 1 was doing wrong. 1 calmed myself down by remembering that students 
would be self-assessing the next day. Hopefully, they would find and fix their 
mistakes at that time. 

The following day 1 briefly reviewed the acronym we were using to remind of us the 
rubric and asked students to give examples for each criterion. The students then 
got to work and worked hard and quietly the entire class period. When 1 graded 
their final essays 1 was shocked, elated, and overwhelmed. The students had risen 
to the challenge. Every single student In the class had scored a three or higher on 
every criterion on the rubric— Including the boy with whom 1 had made the deal. 

What was even more amazing was that the students were just as excited as I was. 

When 1 announced the results the next day, 1 could see the pride they had In their 
accomplishment. They high fived each other and said, "We did It. We actually did It. 

Way to go!" 

This was the best experience of my teaching career thus far. Never before had I felt 
that I had reached the students In a way that let them understand their own 
wonderful accomplishments. 1 also felt reassured that It Is OK to set high 
expectations for students, because they really will rise to the challenge. Above all, 
the best lesson learned was to never give up on students. I always have to keep 
pushing them forward, because It will pay off. 

The Payoff 

Improvements in in-class writing 

Mrs. Buff and the other teachers noticed consistent Improvements In the processes and products 
of students' writing. Mr. Terry, for example, observed the ways In which knowing what counts and 
engaging In self-assessment tended to lead students to work harder at writing well: 

It Is not uncommon to see more students looking for the thesaurus to find that 
enchanting word that they can't seem to put their finger on. For Instance, one of 
my students was writing about horses In a parade. Instead of saying, "The horse 
looked magnificent," she wrote, "The LIpIzzaner stallions were the most magnificent 
horses In the parade." It was gratifying to see the extra effort. 
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Mrs. Erano made similar observations about her eighth grade students' skills and attitudes toward 
writing: 


I believe the rubric helped lessen student resistance to writing and revising. As 
students grow more familiar with the rubric, they seem to better understand how to 
evaluate themselves and each other, as well as how to complete the writing 
process. The rubric is their tool to use whenever they need it. Working with it 
allows them to focus on details that help them to develop, extend, and clarify their 
ideas. 

A self-disciplined attitude is one by-product. At the risk of sounding cliche, I believe 
true learning became their own. As students take responsibility for helping 
themselves and each other, they become more self- directed. Without tools such as 
the rubric and the COACH process, peer editing and self-direction would be almost 
impossible for most of them. The rubric allows students to monitor their own 
progress and achieve to the best of their ability. 

Mrs. Buff reported that, by the end of the school year, students' writing and their ability to self- 
assess had dramatically improved: 

Students are aware of what makes good writing, and they use it in all their ELA 
writing assignments. Most of the students are continuously scoring a level three or 
four on all criteria on the rubric. Students are also able to self-assess 
independently. 

They are aware of their strengths and weaknesses and make noticeable efforts to 
focus on their weaknesses on each new writing assignment. Now that they 
understand the process of self-assessment, they enjoy it and are ready to begin 
even before I am. The self-assessment gives them a sense of independence and 
helps them take ownership of their writing. 

I mprovements in test scores 

Fortunately, the students' new skills appear to have transferred to the extended response portion 
of the ELA test as well. Figure 4 contains the passing ELA scores (level three or four) for 2006 and 
2007. In grades six and eight, the scores for all students, taken together, increased by seven and 
15%, respectively. Subgroup scores also went up. Most impressive are the improvements in 
scores received by economically disadvantaged students, which were 20% higher in both grades. 

Figure 4 

Scores on the English Language Arts test for 2006 and 2007 by grade level and subgroup, and 
between-year differences. 


Grade ievei Subgroup 

2006 

ELA scores 
of 3 or 4 

2007 

ELA scores 
of 3 or 4 

Difference 

Sixth grade 




All students 

50% 

57% 

+1 

African-American 

30% 

46% 

-F16 

Special Needs 

5% 

20% 

-F15 

Econmically Disadvantaged 

28% 

48% 

-F20 

Seventh grade 




All students 

47% 

47% 

0 
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African-American 

35% 

34% 

-1 

Special Needs 

0% 

5% 

-F5 

Econmically Disadvantaged 

31% 

33% 

-F2 

Eighth grade 




All students 

37% 

52% 

-F15 

African-American 

23% 

45% 

-F22 

Special Needs 

3% 

15% 

-F12 

Econmically Disadvantaged 

20% 

40% 

-F20 


The scores for seventh grade were essentially unchanged, probably because the seventh grade 
test does not require students to actually write. In seventh grade, students are asked to answer 
multiple choice questions, write short responses, and edit a passage. Their new skills in writing 
extended responses were not measured. 

Although we were very pleased with the results of the ELA test for the sixth and eighth grades, it 
is important to note that the increased scores cannot be attributed solely to our work. Different 
students were tested each year, so some variation was to be expected, regardless of what we did 
The ELA test scores had been relatively stagnant in previous years, however. The percentages of 
passing scores at eighth grade, for example, were 44%, 35%, 39%, and 37% for the years 2003 
through 2006, respectively. By 2007, 52% of the eighth graders passed the ELA test. Given that 
trend and the improvements in the writing students did in class, we have reason to believe that 
our work had an impact. Our original goals— making assessments clear to students; providing 
frequent feedback about the quality of their work via teacher, peer, and self-assessment; and 
using classroom assessment results to plan instruction— appear to have served student learning 
and school progress. 
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